On Geometric Algebra representation of Binary Spatter Codes 



Diederik Aerts 1 , Marek Czachor 1,2 , and Bart De Moor 3 
1 Centrum Leo Apostel (CLEA) and Foundations of the Exact Sciences (FUND) 
Brussels Free University, 1050 Brussels, Belgium 
2 Katedra Fizyki Teoretycznej i Informatyki Kwantowej 
Politechmka Gdanska, 80-952 Gdansk, Poland 
3 ESAD-SCD, K. U. Leuven, 3001 Leuwen, Belgium 

Kanerva's Binary Spatter Codes are reformulated in terms of geometric algebra. The key ingre- 
dient of the construction is the representation of XOR binding in terms of geometric product. 



I. INTRODUCTION 

\o • 
o 

Distributed representation is a way of representing information in a pattern of activation over a set of neurons, in 
which each concept is represented by activation over multiple neurons, and each neuron participates in the represen- 
^ ' tation of multiple concepts 0] . Examples of distributed representations include Recursive Auto- Associative Memory 
(RAAM) Q , Tensor Product Representations , Holographic Reduced Representations (HRRs) 0, E| , and Binary 
O" Spatter Codes (BSC) H[l||. 

, BSC is a powerful and simple method of representing hierarchical structures in connectionist systems and may be 
■ regarded as a binary version of HRRs. Yet, BSC has some drawbacks associated with the representation of chunking. 
This is why different versions of BSC can be found in the literature. In [(J Q chunking is given by a majority-rule 
thresholded addition of binary strings, an operation that often discards a lot of important information. In y| the 
ordinary addition is employed, and bits are parametrized differently. 

The main message we want to convey in this paper is that there exists a very natural representation of BSC at 



< 



the level of Clifford algebras. Binding of vectors is here performed by means of the Clifford product and chunking is 
just ordinary addition. Since Cliford algebras possess a geometric interpretation in terms of Geometric Algebra (GA) 
fl7l flM llflj. the cognitive structures processed in BSC or HHRs obtain a geometric content. This is philosophically 
consistent with many other approaches where cognition is interpreted in geometric terms 0, |2^. Of particular 
relevance may be the links to neural computation whose GA and HRR versions were formulated by different authors 

; ( C f. UMm 

The present paper can be also seen in a wider context of a "quantum structures" approach to cognitive problems 
we have outlined elsewhere 0, ^3, 0, 0, . Cartan's representation of GA in terms of tensor products of Pauli 
matrices introduces formal links to quantum computation (cf. |H, 0) ES 01 ) • The philosophy we advocate here is 
also not that far from the approach of Widdows, where both geometric an "quantum" aspects play an important role 

It should be stressed that the GA calculus has already proved to be a powerful tool in applied branches of computer 
science (computer vision |20l |. robotics plj)- GA is a comprehensive language that simplified and integrated many 
branches of classical and quantum physics |3l| . One may hope that it will play a similar role in cognitive science. 
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II. BINARY SPATTER CODES 

In BSC information is encoded into long unstructured strings of bits that form a holistic record. The record is 
composed in two steps called binding and chunking. 

Binding of a role x with a filler y is performed by means of XOR © (componentwise addition of binary strings mod 
2); the role-filler object is x © y. Chunking means adding the bound structures in a suitable way. 

In order to illustrate the original BSC and its algebraic modification let us take the example from [7j ■ The encoded 
record is 

PSmith = name © Pat + sex © male + age © 66. (1) 

Decoding of the "name" looks as follows 

Pat' = name © PSmith 

= name © [name © Pat + sex © male + age © 66] 
= Pat + name © sex © male + name © age © 66 

= Pat ©noises Pat. (2) 
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We have used here the involutive nature of XOR and the fact that the "noise" can be eliminated by clean-up memory. 
The latter means that we compare Pat' with records stored in some memory and check, by means of the Hamming 
distance, which of the stored elements is closest to Pat'. A similar trick could be done be means of circular convolution 
in HRRs, but then we would have used an approximate inverse name*, and an appropriate measure of distance. Again, 
the last step is comparison of the noisy object with "pure" objects stored in clean-up memory. 



III. GEOMETRIC-ALGEBRA REPRESENTATION OF BINARY SPATTER CODES 



Euclidean-space GA is constructed as follows. One takes an n-dimensional linear space with orthonormal basis 
{ei, . . . , e„}. Directed subspaces are then associated with the set 



{l,ei 



j e n , ei2, ei3 . 



! ^n- l.Ti) ■ • * ) &12...n\ • 



(3) 



Here 1 corresponds to scalars, i.e. a O-dimensional space. Then we have vectors (oriented segments), bivectors 
(oriented parallelograms), and so on. There exists a natural parametrization: 1 = eo...Oj e± = eio...Oi e 2 = eoio...Oj ■ ■ ■ , 
ei25 = enooio...Oj . . . , ei2... n -i,n — e n...ii which shows that there is a one-to-one relation between an n-bit number 
and an element of GA. An element with k Is and n — k Os is called a /c-blade. 

A geometric product of k 1-blades is a fc-blade. For example, ei248 = eie^^es- Moreover, e n e m — —e m e n , if m =/= n, 
and e n e n = 1, for any n. GA is a Clifford algebra |28| enriched by certain geometric interpretations and operations. 

Particularly interesting is the form of the geometric product that occurs in the binary parametrization. Let us work 
out a few examples: 



eid 
eiei2 

ei257C26 



eio...oeio...o = 1 = eo...o = e(i ...o)e(io...o) (4) 

eio...oeno...o = eieie2 = e 2 = e io...o = e (io...o)©(no...o) (5) 

eno...oeio...o = eie 2 ei = — e 2 eiei = —e 2 = — eoio...o = ^ e (no...o)©(io...o) (6) 

enooioio...oeoioooio...o = eie2e5e7e2e6 = (-I) 2 eie2e 2 e 5 e7e6 = (-1) 2 (-I) 1 eie2e2e5e6e7 

= (-I) 3 eie5e6e7 = (-l) 3 eiooomo...o = (-1) e( 1100 ioio...o)©(oioooio...o) ■ (?) 

The number D is the number of times a 1 from the right string had to "jump" over a 1 from the left one during the 
process of shifting the right string to the left. Symbolically the operation can be represented as 



01000100... 



11001010... 



(-1) J 



01000100. 
11001010. 



(-1) J 



The above observations, generalized to arbitrary strings of bits, yield 

e Al ...A n e Bl ...B n = (-l) 5 ^' 3 *^^. 
Indeed, for two arbitrary strings of bits we have 



01000100. . 

© 

11001010. . 

„)©(£?!. ..£!„)• 



= {-1) D [ 10001110... 0] 



(8) 



B\B 2 



Br, 



AiA 2 



where 

D = B 1 {A 2 
We conclude that the map 



.Ar 



■A„ 



(-1) 



D 



B\B 2 
A X A 2 



■ B n 

■ An 



B 2 (A 3 + ...+An) + --- + Bn-lA. = BkAl 



(9) 



(10) 



k<l 



(Ai . . . An) x (Bi . . . B n ) i-> (A± . . . An) © (Bi . . . B n ) (11) 

has GA projective (i.e. up to a sign) representation by means of JSj. Accordingly, the geometric product is a 
representation of Kanerva's binding at the level of GA. 

Chunking can be represented in GA similarly to what is done in HRRs, that is, by ordinary addition. To see this 
consider 



X — eA 1 ...A n eB 1 ...B n + ec 1 ...c n e D 1 ...D n , 

Y = {-l) E *<i AhA, e Ax ... An X = e Bl ...B n + {-l) T ' k < lAkAl e A] v .tr s < </>,.../- 

= e Bl ...B n ± e(Ai...A n )®(C x ...C n )®(D 1 ...D n ) 

= z Bl ...B„ + noise. 



(12) 
(13) 
(14) 
(15) 
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Until now the procedure is similar to what is done in BSC and HRRs. 

An analogue of clean-up memory can be constructed in various ways. One possibility is to make sure that fillers, 
e-Bx-.-Bn e t c - are orthogonal to the noise term. For example, let us take the fillers of the form eB 1 ...B k o...o, where the 
first k <C n bits are selected at random, but the remaining n — k bits are all 0. Let the roles be taken, as in Kanerva's 
BSC, with all the bits generated at random. The term e(Ai...A n )®(Ci...c n )@(Di...D n ) w iU with high probability contain 
at least one Bj = 1, k < j < n, and thus will be orthogonal to the fillers. The clean-up memory will consist of vectors 
with Bj = 0, k < j < n, i.e of the filler form. 

Tie final step is performed again in analogy to HRRs. We compute a scalar product between Y and the elements 
of clean-up memory. Depending on our needs we can play with different scalar products, or with the so-called 
contractions [2?]]. The richness of GA opens here several possibilities. 

IV. CARTAN REPRESENTATION 

In this section we give an explicit matrix representation of GA. We begin with Pauli's matrices 

*i = I in ) ' ff 2 = f °- n * ) , a 3 = ( \ °, J • (16) 



v 1 J ' A \i J 5 J ^ -1 / 

GA of a plane is represented as follows: 1 = 2x2 unit matrix, e% — ax, e 2 = 02, ei 2 = 0102 = 103. Alternatively, we 
can write eoo = 1, eio = 0i, eoi = 02, en = 103, and 

aooeoo + a w e w + a m e m + a X xexx =[ aw+ t0iQ1 a(]Q _ ian ) ■ (17) 

This is equivalent to encoding 2 2 = 4 real numbers into two complex numbers. 

In 3-dimensional space we have 1 = 2x2 unit matrix, e\ — ax, £2 — 02, £3 — 03, &X2 = 0102 = ^03, ei3 = 0103 = 
— ia%, e 2 3 = 0203 = *0i, ei23 = 010203 = i- 

Now the representation of 

E_( cxooo + ioLxu + aooi + «*uoj aioo + «aoii - «a io - "ioi \ ,,o\ 
abc abc [ + i aQ11 + ia mo + axox, cxqoo + iaxxx — a 00 x - iaxxo ' 



ABC=Q.l 



is equivalent to encoding 2 3 = 8 real numbers into 4 complex numbers. 

An arbitrary n-bit record can be encoded into the matrix algebra known as Cartan's representation of Clifford 
algebras [H: 

e 2 k = 01 ® ■ ■ ■ ® 01 02 <S> 1 ® ■ ■ ■ g 1 , (19) 
n—k fc-1 
e 2 fe_i = 01 8) ■ ■ ■ 8 01 O 03 ® 1 (E> ■ • • ® 1 ■ (20) 

n— fe fe— 1 

In practical calculations it is convenient to work with the tensor product implemented by means of the "drag-and-drop" 
rule: For arbitrary matrices A and B (not necessarily square, and possibly of different dimensions) 



A®B 



/an ■■■ axj \ ( bxx ■■■ bx s \ ( axxB ... axjB 



\ a>n ■ ■ ■ a>ij ) \ b r x ■ ■ ■ b rs f \a%xB . . . aijB 



( axxbxx ■ ■ ■ axjbxs 



(21) 



\ OiixWl ■ ■ ■ ttijb rs 



This representation of <g> can be used to show that all given by Cartan's representation are matrices of zero trace. 

V. PAT SMITH REVISITED 

So let us return to the example from Sec. 2. For simplicity take n — 4 so that we can choose the representation 

epat = enoo, 

e ma ic = eiooo, }• fillers (22) 

666 = eoiOCb 
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^namc ~~ 

e sox = e in, } roles (23) 

^age — 

The fillers have only the first two bits selected at random, the last two are 00. The roles are numbered by randomly 
selected strings of bits. 

The explicit matrix representations are: 



epat 


= enoo 


= eie 2 


= (fTi ® 


01 (8 01 <8 CT 3 )(C71 <8 (Tl § 


5 0i (8 02) = 1 <8 1 <8 1 5 


5 er 3 cr 2 = 1 <8 1 (8 1 (8 (-«0i) (24) 


^malc 


= eiooo 


= ei = 


01 <g) (Ti 


(8 CTi (8 3 




(25) 


eee 


= eoioo 


= e 2 = 


I7i (8) (Tx 


<8> 0i <8> 02 




(26) 


^namc 


= e 1010 


= eie 3 


= (CTJ ® 


0i (8 0i (8 3 )(0i <8 0i << 


5 3 (8 1) 






= 1®1 


<g> 0i0 3 


(8 3 = 


1 (8 1 <8 (-«02) <8 3 







e SC x = eom = e 2 e 3 e4 = (cti (8 01 (8 01 (8 02X01 <8 01 (8 3 (8 l)(0i ® 01 (8> 02 <8> 1) 

= (01 (8 01 (8 01 (8 02XI <8 1 (8 3 02 <8 1) = 01 (8 01 (8 0i0 3 02 <8 02 = 01 <8 01 (8 (— il) (8 02, (27) 
e ag o = eion = eie 3 e4 = (cti (8 01 <8 01 <8 3 )(0i 18 01 <8 3 (8 l)(0i ® 01 <S> 02 <8 1) 

= 01 (801 (8 (-il) (8 03 (28) 
The whole record 

PSmith = ae n am epat + /3e scx e m aie + 7 e a go e66 

= aeioioenoo + /3eomeiooo + 7 e ion e oioo 

= a(-l) 2 e ( i i ) e (ii 00 ) + /3(-l) 3 e (0 iii)e(iooo) + 7(-l) 2e (ion)e(oioo) 
= aeono - /3eim + 7 e im 

The fact that the last two terms are linearly dependent is a consequence of too small dimensionality of our binary 
strings (four bits, whereas in realistic cases Kanerva suggested 10 4 bit strings). This is the price we pay for simplicity 
of the example. Decoding the name involves two steps. First 

enamoPSmith = eioioPSmith = eioio[ae iio - /3eim +7emi] 

= a(-l) 1 e(ioi ) ffi ( ii ) - /3(-l) 2 e(ioio)©(mi) + 7(-l) 2e (ioio)e(mi) 
= -aenoo - pe wi + 7 e oioi 

= -aepat -/3e ioi + 7e i i = Pat'. (29) 

noise 

It remains to employ clean-up memory. But this is easy since the noise is perpendicular to ep a t. We only have to 
project on the set spanned by the fillers, and within this set check which element is closest to the cleaned up Pat'. 

Cartan's represenation allows to define scalar product in GA by means of the trace. We therefore compare scalar 
products between Pat' and elements of clean-up memory. The only nonzero scalar product is 



(e Pat |Pat') = Tr {e xim [ - aenoo - /9e ioi + 7eoioi] ) 

= Tr (-aeie 2 eie 2 - /3eie2e 2 e 4 + 7eie 2 e 2 e 4 ) 
= Tr (al — /3eie4 + 7eie4) = 16a. 



(30) 



Indeed 

eie 4 = eioooeoooi = eiooi = (01 ® 01 <S> 01 <S> 03X01 <8 01 <8 02 <8 1) = 1 <8 1 <8 («0 3 ) ® 3 

= diag (i, — i, — i, i, i, — i, — i, i, i, — i, — i, i, i, — i, — i, i) (31) 

has zero trace. 

VI. CONCLUSIONS 

BSC represented at the level of GA maintain the essential element of the original construction, i.e. binding by 
means of XOR. However, instead of the straightforward map 

(x,y)^x©y (32) 
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we rather have the "exponential map" x i— > e x satisfying e x e y = ±e x ® y . Another difference is in mathematical 
implementation of chunking. Unbinding produces a noise term which, with high probability, is orthogonal to the 
original filler. In this respect the construction is analogous to error correcting linear codes. As opposed to tensor 
product representations, and similarly to BSC and HRRs, binding performed by means of geometric product does not 
increase dimensions. 
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